On some document clustering algorithms for data mining

We consider the problem of clustering large document sets into disjoint groups or clusters. Our starting point is recent literature on effective clustering algorithms, specifically Principal Direction Divisive Partitioning (PDDP), proposed by Boley in [1] and Spherical k-Means (“S–kmeans” for short) proposed by Dhillon and Moda in [4]. In this paper we study and evaluate the performance of these algorithms and propose specific refinements. We also explore the effectiveness of PDDP for various partitioning and termination rules. Finally we present results that demonstrate the effectiveness of both PDDP and Skmeans, for the computation of low rank matrix approximations. Document clustering is heavily used in many fields including data mining and information retrieval. The vector space model in document clustering uses an m×n, e.g. (term) by (document) or (term frequency by document) matrix, where m is the number of terms or attributes and n the number of documents. Clustering algorithms can be divided into “partitional” and “hierarchical”. Examples of the former are the k-means algorithm [5] as well as its aforementioned variation, S–kmeans. The latter, hierarchical class includes algorithms that produce clusters via a recursive “agglomerative” (bottom-up) or “divisive” (top-down) process. A recent effective divisive algorithm from this class is PDDP. k-means and its aforementioned variant are easy to implement and appear to lend themselves well to parallelization [3]. On the other hand, they are prone to converge to solutions that are only locally optimal (finding a global minimum is NP-complete) and